Programming with libbzip2

This chapter describes the programming interface to libbzip2.

For general background information, particularly about memory use and performance aspects, you'd be well advised to read Chapter 2 as well.

Top-level structure

libbzip2 is a flexible library for compressing and decompressing data in the bzip2 data format. Although packaged as a single entity, it helps to regard the library as three separate parts: the low level interface, and the high level interface, and some utility functions.

The structure of libbzip2's interfaces is similar to that of Jean-loup Gailly's and Mark Adler's excellent zlib library.

Low-level summary

This interface provides services for compressing and decompressing data in memory. There's no provision for dealing with files, streams or any other I/O mechanisms, just straight memory-to-memory work. In fact, this part of the library can be compiled without inclusion of stdio.h, which may be helpful for embedded applications.

The low-level part of the library has no global variables and is therefore thread-safe.

Six routines make up the low level interface: bzCompressInit, bzCompress, and
bzCompressEnd for compression, and a corresponding trio bzDecompressInit,
bzDecompress and bzDecompressEnd for decompression. The *Init functions allocate memory for compression/decompression and do other initialisations, whilst the *End functions close down operations and release memory.

The real work is done by bzCompress and bzDecompress. These compress/decompress data from a user-supplied input buffer to a user-supplied output buffer. These buffers can be any size; arbitrary quantities of data are handled by making repeated calls to these functions. This is a flexible mechanism allowing a consumer-pull style of activity, or producer-push, or a mixture of both.

High-level summary

This interface provides some handy wrappers around the low-level interface to facilitate reading and writing bzip2 format files (.bz2 files). The routines provide hooks to facilitate reading files in which the bzip2 data stream is embedded within some larger-scale file structure, or where there are multiple bzip2 data streams concatenated end-to-end.

For reading files, bzReadOpen, bzRead, bzReadClose and bzReadGetUnused are supplied. For writing files, bzWriteOpen, bzWrite and bzWriteFinish are available.

As with the low-level library, no global variables are used so the library is per se thread-safe. However, if I/O errors occur whilst reading or writing the underlying compressed files, you may have to consult errno to determine the cause of the error. In that case, you'd need a C library which correctly supports errno in a multithreaded environment.

To make the library a little simpler and more portable, bzReadOpen and bzWriteOpen require you to pass them file handles (FILE*s) which have previously been opened for reading or writing respectively. That avoids portability problems associated with file operations and file attributes, whilst not being much of an imposition on the programmer.

Utility functions summary

For very simple needs, bzBuffToBuffCompress and bzBuffToBuffDecompress are provided. These compress data in memory from one buffer to another buffer in a single function call. You should assess whether these functions fulfill your memory-to-memory compression/decompression requirements before investing effort in understanding the more general but more complex low-level interface.

Yoshioka Tsuneo (QWF00133@niftyserve.or.jp / tsuneo-y@is.aist-nara.ac.jp) has contributed some functions to give better zlib compatibility. These functions are bzopen, bzread, bzwrite, bzflush, bzclose, bzerror and bzlibVersion. You may find these functions more convenient for simple file reading and writing, than those in the high-level interface. These functions are not (yet) officially part of the library, and are not further documented here. If they break, you get to keep all the pieces. I hope to document them properly when time permits.

Yoshioka also contributed modifications to allow the library to be built as a Windows DLL.

Error handling

The library is designed to recover cleanly in all situations, including the worst-case situation of decompressing random data. I'm not 100% sure that it can always do this, so you might want to add a signal handler to catch segmentation violations during decompression if you are feeling especially paranoid. I would be interested in hearing more about the robustness of the library to corrupted compressed data.

The file bzlib.h contains all definitions needed to use the library. In particular, you should definitely not include bzlib_private.h.

In bzlib.h, the various return values are defined. The following list is not intended as an exhaustive description of the circumstances in which a given value may be returned -- those descriptions are given later. Rather, it is intended to convey the rough meaning of each return value. The first five actions are normal and not intended to denote an error situation.

BZ_OK

The requested action was completed successfully.

BZ_RUN_OK

BZ_FLUSH_OK

BZ_FINISH_OK

In bzCompress, the requested flush/finish/nothing-special action was completed successfully.

BZ_STREAM_END

Compression of data was completed, or the logical stream end was detected during decompression.

The following return values indicate an error of some kind.

BZ_SEQUENCE_ERROR

When using the library, it is important to call the functions in the correct sequence and with data structures (buffers etc) in the correct states. libbzip2 checks as much as it can to ensure this is happening, and returns BZ_SEQUENCE_ERROR if not. Code which complies precisely with the function semantics, as detailed below, should never receive this value; such an event denotes buggy code which you should investigate.

BZ_PARAM_ERROR

Returned when a parameter to a function call is out of range or otherwise manifestly incorrect. As with BZ_SEQUENCE_ERROR, this denotes a bug in the client code. The distinction between BZ_PARAM_ERROR and BZ_SEQUENCE_ERROR is a bit hazy, but still worth making.

BZ_MEM_ERROR

Returned when a request to allocate memory failed. Note that the quantity of memory needed to decompress a stream cannot be determined until the stream's header has been read. So bzDecompress and bzRead may return BZ_MEM_ERROR even though some of the compressed data has been read. The same is not true for compression; once bzCompressInit or bzWriteOpen have successfully completed, BZ_MEM_ERROR cannot occur.

BZ_DATA_ERROR

Returned when a data integrity error is detected during decompression. Most importantly, this means when stored and computed CRCs for the data do not match. This value is also returned upon detection of any other anomaly in the compressed data.

BZ_DATA_ERROR_MAGIC

As a special case of BZ_DATA_ERROR, it is sometimes useful to know when the compressed stream does not start with the correct magic bytes ('B' 'Z' 'h').

BZ_IO_ERROR

Returned by bzRead and bzRead when there is an error reading or writing in the compressed file, and by bzReadOpen and bzWriteOpen for attempts to use a file for which the error indicator (viz, ferror(f)) is set. On receipt of BZ_IO_ERROR, the caller should consult errno and/or perror to acquire operating-system specific information about the problem.

BZ_UNEXPECTED_EOF

Returned by bzRead when the compressed file finishes before the logical end of stream is detected.

BZ_OUTBUFF_FULL

Returned by bzBuffToBuffCompress and bzBuffToBuffDecompress to indicate that the output data will not fit into the output buffer provided.

Low-level interface

bzCompressInit

typedef 
   struct {
      char *next_in;
      unsigned int avail_in;
      unsigned int total_in;

      char *next_out;
      unsigned int avail_out;
      unsigned int total_out;

      void *state;

      void *(*bzalloc)(void *,int,int);
      void (*bzfree)(void *,void *);
      void *opaque;
   } 
   bz_stream;

int bzCompressInit ( bz_stream *strm, 
                     int blockSize100k, 
                     int verbosity,
                     int workFactor );

Prepares for compression. The bz_stream structure holds all data pertaining to the compression activity. A bz_stream structure should be allocated and initialised prior to the call. The fields of bz_stream comprise the entirety of the user-visible data. state is a pointer to the private data structures required for compression.

Custom memory allocators are supported, via fields bzalloc, bzfree, and opaque. The value opaque is passed to as the first argument to all calls to bzalloc and bzfree, but is otherwise ignored by the library. The call bzalloc ( opaque, n, m ) is expected to return a pointer p to n * m bytes of memory, and bzfree ( opaque, p ) should free that memory.

If you don't want to use a custom memory allocator, set bzalloc, bzfree and opaque to NULL, and the library will then use the standard malloc/free routines.

Before calling bzCompressInit, fields bzalloc, bzfree and opaque should be filled appropriately, as just described. Upon return, the internal state will have been allocated and initialised, and total_in and total_out will have been set to zero. These last two fields are used by the library to inform the caller of the total amount of data passed into and out of the library, respectively. You should not try to change them.

Parameter blockSize100k specifies the block size to be used for compression. It should be a value between 1 and 9 inclusive, and the actual block size used is 100000 x this figure. 9 gives the best compression but takes most memory.

Parameter verbosity should be set to a number between 0 and 4 inclusive. 0 is silent, and greater numbers give increasingly verbose monitoring/debugging output. If the library has been compiled with -DBZ_NO_STDIO, no such output will appear for any verbosity setting.

Parameter workFactor controls how the compression phase behaves when presented with worst case, highly repetitive, input data. If compression runs into difficulties caused by repetitive data, some pseudo-random variations are inserted into the block, and compression is restarted. Lower values of workFactor reduce the tolerance of compression to repetitive data. You should set this parameter carefully; too low, and compression ratio suffers, too high, and your average-to-worst case compression times can become very large. The default value of 30 gives reasonable behaviour over a wide range of circumstances.

Allowable values range from 0 to 250 inclusive. 0 is a special case, equivalent to using the default value of 30.

Note that the randomisation process is entirely transparent. If the library decides to randomise and restart compression on a block, it does so without comment. Randomised blocks are automatically de-randomised during decompression, so data integrity is never compromised.

Possible return values:

      BZ_PARAM_ERROR 
         if strm is NULL 
         or blockSize < 1 or blockSize > 9
         or verbosity < 0 or verbosity > 4
         or workFactor < 0 or workFactor > 250
      BZ_MEM_ERROR 
         if not enough memory is available
      BZ_OK 
         otherwise

Allowable next actions:

      bzCompress 
         if BZ_OK is returned
      no specific action needed in case of error

bzCompress

   int bzCompress ( bz_stream *strm, int action );

Provides more input and/or output buffer space for the library. The caller maintains input and output buffers, and calls bzCompress to transfer data between them.

Before each call to bzCompress, next_in should point at the data to be compressed, and avail_in should indicate how many bytes the library may read. bzCompress updates next_in, avail_in and total_in to reflect the number of bytes it has read.

Similarly, next_out should point to a buffer in which the compressed data is to be placed, with avail_out indicating how much output space is available. bzCompress updates next_out, avail_out and total_out to reflect the number of bytes output.

You may provide and remove as little or as much data as you like on each call of bzCompress. In the limit, it is acceptable to supply and remove data one byte at a time, although this would be terribly inefficient. You should always ensure that at least one byte of output space is available at each call.

A second purpose of bzCompress is to request a change of mode of the compressed stream.

Conceptually, a compressed stream can be in one of four states: IDLE, RUNNING, FLUSHING and FINISHING. Before initialisation (bzCompressInit) and after termination (bzCompressEnd), a stream is regarded as IDLE.

Upon initialisation (bzCompressInit), the stream is placed in the RUNNING state. Subsequent calls to bzCompress should pass BZ_RUN as the requested action; other actions are illegal and will result in BZ_SEQUENCE_ERROR.

At some point, the calling program will have provided all the input data it wants to. It will then want to finish up -- in effect, asking the library to process any data it might have buffered internally. In this state, bzCompress will no longer attempt to read data from next_in, but it will want to write data to next_out. Because the output buffer supplied by the user can be arbitrarily small, the finishing-up operation cannot necessarily be done with a single call of bzCompress.

Instead, the calling program passes BZ_FINISH as an action to bzCompress. This changes the stream's state to FINISHING. Any remaining input (ie, next_in[0 .. avail_in-1]) is compressed and transferred to the output buffer. To do this, bzCompress must be called repeatedly until all the output has been consumed. At that point, bzCompress returns BZ_STREAM_END, and the stream's state is set back to IDLE. bzCompressEnd should then be called.

Just to make sure the calling program does not cheat, the library makes a note of avail_in at the time of the first call to bzCompress which has BZ_FINISH as an action (ie, at the time the program has announced its intention to not supply any more input). By comparing this value with that of avail_in over subsequent calls to bzCompress, the library can detect any attempts to slip in more data to compress. Any calls for which this is detected will return BZ_SEQUENCE_ERROR. This indicates a programming mistake which should be corrected.

Instead of asking to finish, the calling program may ask bzCompress to take all the remaining input, compress it and terminate the current (Burrows-Wheeler) compression block. This could be useful for error control purposes. The mechanism is analogous to that for finishing: call bzCompress with an action of BZ_FLUSH, remove output data, and persist with the BZ_FLUSH action until the value BZ_RUN is returned. As with finishing, bzCompress detects any attempt to provide more input data once the flush has begun.

Once the flush is complete, the stream returns to the normal RUNNING state.

This all sounds pretty complex, but isn't really. Here's a table which shows which actions are allowable in each state, what action will be taken, what the next state is, and what the non-error return values are. Note that you can't explicitly ask what state the stream is in, but nor do you need to -- it can be inferred from the values returned by bzCompress.

IDLE/any           
      Illegal.  IDLE state only exists after bzCompressEnd or
      before bzCompressInit.
      Return value = BZ_SEQUENCE_ERROR

RUNNING/BZ_RUN     
      Compress from next_in to next_out as much as possible.
      Next state = RUNNING
      Return value = BZ_RUN_OK

RUNNING/BZ_FLUSH   
      Remember current value of next_in.  Compress from next_in
      to next_out as much as possible, but do not accept any more input.  
      Next state = FLUSHING
      Return value = BZ_FLUSH_OK

RUNNING/BZ_FINISH  
      Remember current value of next_in.  Compress from next_in
      to next_out as much as possible, but do not accept any more input.
      Next state = FINISHING
      Return value = BZ_FINISH_OK

FLUSHING/BZ_FLUSH  
      Compress from next_in to next_out as much as possible, 
      but do not accept any more input.  
      If all the existing input has been used up and all compressed
      output has been removed
         Next state = RUNNING; Return value = BZ_RUN_OK
      else
         Next state = FLUSHING; Return value = BZ_FLUSH_OK

FLUSHING/other     
      Illegal.
      Return value = BZ_SEQUENCE_ERROR

FINISHING/BZ_FINISH  
      Compress from next_in to next_out as much as possible,
      but to not accept any more input.  
      If all the existing input has been used up and all compressed
      output has been removed
         Next state = IDLE; Return value = BZ_STREAM_END
      else
         Next state = FINISHING; Return value = BZ_FINISHING

FINISHING/other
      Illegal.
      Return value = BZ_SEQUENCE_ERROR

That still looks complicated? Well, fair enough. The usual sequence of calls for compressing a load of data is:

• Get started with bzCompressInit.
• Shovel data in and shlurp out its compressed form using zero or more calls of bzCompress with action = BZ_RUN.
• Finish up. Repeatedly call bzCompress with action = BZ_FINISH, copying out the compressed output, until BZ_STREAM_END is returned.
• Close up and go home. Call bzCompressEnd.

If the data you want to compress fits into your input buffer all at once, you can skip the calls of bzCompress ( ..., BZ_RUN ) and just do the bzCompress ( ..., BZ_FINISH ) calls.

All required memory is allocated by bzCompressInit. The compression library can accept any data at all (obviously). So you shouldn't get any error return values from the bzCompress calls. If you do, they will be BZ_SEQUENCE_ERROR, and indicate a bug in your programming.

Trivial other possible return values:

      BZ_PARAM_ERROR   
         if strm is NULL, or strm->s is NULL

bzCompressEnd

int bzCompressEnd ( bz_stream *strm );

Releases all memory associated with a compression stream.

Possible return values:

   BZ_PARAM_ERROR    if strm is NULL or strm->s is NULL
   BZ_OK    otherwise

bzDecompressInit

int bzDecompressInit ( bz_stream *strm, int verbosity, int small );

Prepares for decompression. As with bzCompressInit, a bz_stream record should be allocated and initialised before the call. Fields bzalloc, bzfree and opaque should be set if a custom memory allocator is required, or made NULL for the normal malloc/free routines. Upon return, the internal state will have been initialised, and total_in and total_out will be zero.

For the meaning of parameter verbosity, see bzCompressInit.

If small is nonzero, the library will use an alternative decompression algorithm which uses less memory but at the cost of decompressing more slowly (roughly speaking, half the speed, but the maximum memory requirement drops to around 2300k). See Chapter 2 for more information on memory management.

Note that the amount of memory needed to decompress a stream cannot be determined until the stream's header has been read, so even if bzDecompressInit succeeds, a subsequent bzDecompress could fail with BZ_MEM_ERROR.

Possible return values:

      BZ_PARAM_ERROR
         if (small != 0 && small != 1)
         or (verbosity < 0 || verbosity > 4)
      BZ_MEM_ERROR
         if insufficient memory is available

Allowable next actions:

      bzDecompress
         if BZ_OK was returned
      no specific action required in case of error

bzDecompress

int bzDecompress ( bz_stream *strm );

Provides more input and/out output buffer space for the library. The caller maintains input and output buffers, and uses bzDecompress to transfer data between them.

Before each call to bzDecompress, next_in should point at the compressed data, and avail_in should indicate how many bytes the library may read. bzDecompress updates next_in, avail_in and total_in to reflect the number of bytes it has read.

Similarly, next_out should point to a buffer in which the uncompressed output is to be placed, with avail_out indicating how much output space is available. bzCompress updates next_out, avail_out and total_out to reflect the number of bytes output.

You may provide and remove as little or as much data as you like on each call of bzDecompress. In the limit, it is acceptable to supply and remove data one byte at a time, although this would be terribly inefficient. You should always ensure that at least one byte of output space is available at each call.

Use of bzDecompress is simpler than bzCompress.

You should provide input and remove output as described above, and repeatedly call bzDecompress until BZ_STREAM_END is returned. Appearance of BZ_STREAM_END denotes that bzDecompress has detected the logical end of the compressed stream. bzDecompress will not produce BZ_STREAM_END until all output data has been placed into the output buffer, so once BZ_STREAM_END appears, you are guaranteed to have available all the decompressed output, and bzDecompressEnd can safely be called.

If case of an error return value, you should call bzDecompressEnd to clean up and release memory.

Possible return values:

      BZ_PARAM_ERROR
         if strm is NULL or strm->s is NULL
         or strm->avail_out < 1
      BZ_DATA_ERROR
         if a data integrity error is detected in the compressed stream
      BZ_DATA_ERROR_MAGIC
         if the compressed stream doesn't begin with the right magic bytes
      BZ_MEM_ERROR
         if there wasn't enough memory available
      BZ_STREAM_END
         if the logical end of the data stream was detected and all
         output in has been consumed, eg s->avail_out > 0
      BZ_OK
         otherwise

Allowable next actions:

      bzDecompress
         if BZ_OK was returned
      bzDecompressEnd
         otherwise

bzDecompressEnd

int bzDecompressEnd ( bz_stream *strm );

Releases all memory associated with a decompression stream.

Possible return values:

      BZ_PARAM_ERROR
         if strm is NULL or strm->s is NULL
      BZ_OK
         otherwise

Allowable next actions:

      None.

High-level interface

This interface provides functions for reading and writing bzip2 format files. First, some general points.

• All of the functions take an int* first argument, bzerror. After each call, bzerror should be consulted first to determine the outcome of the call. If bzerror is BZ_OK, the call completed successfully, and only then should the return value of the function (if any) be consulted. If bzerror is BZ_IO_ERROR, there was an error reading/writing the underlying compressed file, and you should then consult errno/perror to determine the cause of the difficulty. bzerror may also be set to various other values; precise details are given on a per-function basis below.
• If bzerror indicates an error (ie, anything except BZ_OK and BZ_STREAM_END), you should immediately call bzReadClose (or bzWriteClose, depending on whether you are attempting to read or to write) to free up all resources associated with the stream. Once an error has been indicated, behaviour of all calls except bzReadClose (bzWriteClose) is undefined. The implication is that (1) bzerror should be checked after each call, and (2) if bzerror indicates an error, bzReadClose (bzWriteClose) should then be called to clean up.
• The FILE* arguments passed to bzReadOpen/bzWriteOpen should be set to binary mode. Most Unix systems will do this by default, but other platforms, including Windows and Mac, will not. If you omit this, you may encounter problems when moving code to new platforms.
• Memory allocation requests are handled by malloc/free. At present there is no facility for user-defined memory allocators in the file I/O functions (could easily be added, though).

bzReadOpen

   typedef void BZFILE;

   BZFILE *bzReadOpen ( int *bzerror, FILE *f, 
                        int small, int verbosity,
                        void *unused, int nUnused );

Prepare to read compressed data from file handle f. f should refer to a file which has been opened for reading, and for which the error indicator (ferror(f))is not set. If small is 1, the library will try to decompress using less memory, at the expense of speed.

For reasons explained below, bzRead will decompress the nUnused bytes starting at unused, before starting to read from the file f. At most BZ_MAX_UNUSED bytes may be supplied like this. If this facility is not required, you should pass NULL and 0 for unused and nUnused respectively.

For the meaning of parameters small and verbosity, see bzDecompressInit.

The amount of memory needed to decompress a file cannot be determined until the file's header has been read. So it is possible that bzReadOpen returns BZ_OK but a subsequent call of bzRead will return BZ_MEM_ERROR.

Possible assignments to bzerror:

      BZ_PARAM_ERROR
         if f is NULL 
         or small is neither 0 nor 1                 
         or (unused == NULL && nUnused != 0)
         or (unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED))
      BZ_IO_ERROR    
         if ferror(f) is nonzero
      BZ_MEM_ERROR   
         if insufficient memory is available
      BZ_OK
         otherwise.

Possible return values:

      Pointer to an abstract BZFILE        
         if bzerror is BZ_OK   
      NULL
         otherwise

Allowable next actions:

      bzRead
         if bzerror is BZ_OK   
      bzClose 
         otherwise

bzRead

   int bzRead ( int *bzerror, BZFILE *b, void *buf, int len );

Reads up to len (uncompressed) bytes from the compressed file b into the buffer buf. If the read was successful, bzerror is set to BZ_OK and the number of bytes read is returned. If the logical end-of-stream was detected, bzerror will be set to BZ_STREAM_END, and the number of bytes read is returned. All other bzerror values denote an error.

bzRead will supply len bytes, unless the logical stream end is detected or an error occurs. Because of this, it is possible to detect the stream end by observing when the number of bytes returned is less than the number requested. Nevertheless, this is regarded as inadvisable; you should instead check bzerror after every call and watch out for BZ_STREAM_END.

Internally, bzRead copies data from the compressed file in chunks of size BZ_MAX_UNUSED bytes before decompressing it. If the file contains more bytes than strictly needed to reach the logical end-of-stream, bzRead will almost certainly read some of the trailing data before signalling BZ_SEQUENCE_END. To collect the read but unused data once BZ_SEQUENCE_END has appeared, call bzReadGetUnused immediately before bzReadClose.

Possible assignments to bzerror:

      BZ_PARAM_ERROR
         if b is NULL or buf is NULL or len < 0
      BZ_SEQUENCE_ERROR 
         if b was opened with bzWriteOpen
      BZ_IO_ERROR 
         if there is an error reading from the compressed file
      BZ_UNEXPECTED_EOF 
         if the compressed file ended before the logical end-of-stream was detected
      BZ_DATA_ERROR 
         if a data integrity error was detected in the compressed stream
      BZ_DATA_ERROR_MAGIC
         if the stream does not begin with the requisite header bytes (ie, is not 
         a bzip2 data file).  This is really a special case of BZ_DATA_ERROR.
      BZ_MEM_ERROR 
         if insufficient memory was available
      BZ_STREAM_END 
         if the logical end of stream was detected.
      BZ_OK
         otherwise.

Possible return values:

      number of bytes read
         if bzerror is BZ_OK or BZ_STREAM_END
      undefined
         otherwise

Allowable next actions:

      collect data from buf, then bzRead or bzReadClose
         if bzerror is BZ_OK 
      collect data from buf, then bzReadClose or bzReadGetUnused 
         if bzerror is BZ_SEQUENCE_END   
      bzReadClose 
         otherwise

bzReadGetUnused

   void bzReadGetUnused ( int* bzerror, BZFILE *b, 
                          void** unused, int* nUnused );

Returns data which was read from the compressed file but was not needed to get to the logical end-of-stream. *unused is set to the address of the data, and *nUnused to the number of bytes. *nUnused will be set to a value between 0 and BZ_MAX_UNUSED inclusive.

This function may only be called once bzRead has signalled BZ_STREAM_END but before bzReadClose.

Possible assignments to bzerror:

      BZ_PARAM_ERROR 
         if b is NULL 
         or unused is NULL or nUnused is NULL
      BZ_SEQUENCE_ERROR 
         if BZ_STREAM_END has not been signalled
         or if b was opened with bzWriteOpen
     BZ_OK
         otherwise

Allowable next actions:

      bzReadClose

bzReadClose

   void bzReadClose ( int *bzerror, BZFILE *b );

Releases all memory pertaining to the compressed file b. bzReadClose does not call fclose on the underlying file handle, so you should do that yourself if appropriate. bzReadClose should be called to clean up after all error situations.

Possible assignments to bzerror:

      BZ_SEQUENCE_ERROR 
         if b was opened with bzOpenWrite 
      BZ_OK 
         otherwise

Allowable next actions:

      none

bzWriteOpen

   BZFILE *bzWriteOpen ( int *bzerror, FILE *f, 
                         int blockSize100k, int verbosity,
                         int workFactor );

Prepare to write compressed data to file handle f. f should refer to a file which has been opened for writing, and for which the error indicator (ferror(f))is not set.

For the meaning of parameters blockSize100k, verbosity and workFactor, see
bzCompressInit.

All required memory is allocated at this stage, so if the call completes successfully, BZ_MEM_ERROR cannot be signalled by a subsequent call to bzWrite.

Possible assignments to bzerror:

      BZ_PARAM_ERROR 
         if f is NULL 
         or blockSize100k < 1 or blockSize100k > 9
      BZ_IO_ERROR 
         if ferror(f) is nonzero
      BZ_MEM_ERROR 
         if insufficient memory is available
      BZ_OK 
         otherwise

Possible return values:

      Pointer to an abstract BZFILE  
         if bzerror is BZ_OK   
      NULL 
         otherwise

Allowable next actions:

      bzWrite 
         if bzerror is BZ_OK 
         (you could go directly to bzWriteClose, but this would be pretty pointless)
      bzWriteClose 
         otherwise

bzWrite

   void bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );

Absorbs len bytes from the buffer buf, eventually to be compressed and written to the file.

Possible assignments to bzerror:

      BZ_PARAM_ERROR 
         if b is NULL or buf is NULL or len < 0
      BZ_SEQUENCE_ERROR 
         if b was opened with bzReadOpen
      BZ_IO_ERROR 
         if there is an error writing the compressed file.
      BZ_OK 
         otherwise

bzWriteClose

   int bzWriteClose ( int *bzerror, BZFILE* f,
                      int abandon,
                      unsigned int* nbytes_in,
                      unsigned int* nbytes_out );

Compresses and flushes to the compressed file all data so far supplied by bzWrite. The logical end-of-stream markers are also written, so subsequent calls to bzWrite are illegal. All memory associated with the compressed file b is released. fflush is called on the compressed file, but it is not fclose'd.

If bzWriteClose is called to clean up after an error, the only action is to release the memory. The library records the error codes issued by previous calls, so this situation will be detected automatically. There is no attempt to complete the compression operation, nor to fflush the compressed file. You can force this behaviour to happen even in the case of no error, by passing a nonzero value to abandon.

If nbytes_in is non-null, *nbytes_in will be set to be the total volume of uncompressed data handled. Similarly, nbytes_out will be set to the total volume of compressed data written.

Possible assignments to bzerror:

      BZ_SEQUENCE_ERROR 
         if b was opened with bzReadOpen
      BZ_IO_ERROR 
         if there is an error writing the compressed file
      BZ_OK 
         otherwise

Handling embedded compressed data streams

The high-level library facilitates use of bzip2 data streams which form some part of a surrounding, larger data stream.

• For writing, the library takes an open file handle, writes compressed data to it, fflushes it but does not fclose it. The calling application can write its own data before and after the compressed data stream, using that same file handle.
• Reading is more complex, and the facilities are not as general as they could be since generality is hard to reconcile with efficiency. bzRead reads from the compressed file in blocks of size BZ_MAX_UNUSED bytes, and in doing so probably will overshoot the logical end of compressed stream. To recover this data once decompression has ended, call bzReadGetUnused after the last call of bzRead (the one returning BZ_STREAM_END) but before calling bzReadClose.

This mechanism makes it easy to decompress multiple bzip2 streams placed end-to-end. As the end of one stream, when bzRead returns BZ_STREAM_END, call bzReadGetUnused to collect the unused data (copy it into your own buffer somewhere). That data forms the start of the next compressed stream. To start uncompressing that next stream, call bzReadOpen again, feeding in the unused data via the unused/nUnused parameters. Keep doing this until BZ_STREAM_END return coincides with the physical end of file (feof(f)). In this situation bzReadGetUnused will of course return no data.

This should give some feel for how the high-level interface can be used. If you require extra flexibility, you'll have to bite the bullet and get to grips with the low-level interface.

Standard file-reading/writing code

Here's how you'd write data to a compressed file:

FILE*   f;
BZFILE* b;
int     nBuf;
char    buf[ /* whatever size you like */ ];
int     bzerror;
int     nWritten;

f = fopen ( "myfile.bz2", "w" );
if (!f) {
   /* handle error */
}
b = bzWriteOpen ( &bzerror, f, 9 );
if (bzerror != BZ_OK) {
   bzWriteClose ( b );
   /* handle error */
}

while ( /* condition */ ) {
   /* get data to write into buf, and set nBuf appropriately */
   nWritten = bzWrite ( &bzerror, b, buf, nBuf );
   if (bzerror == BZ_IO_ERROR) { 
      bzWriteClose ( &bzerror, b );
      /* handle error */
   }
}

bzWriteClose ( &bzerror, b );
if (bzerror == BZ_IO_ERROR) {
   /* handle error */
}

And to read from a compressed file:

FILE*   f;
BZFILE* b;
int     nBuf;
char    buf[ /* whatever size you like */ ];
int     bzerror;
int     nWritten;

f = fopen ( "myfile.bz2", "r" );
if (!f) {
   /* handle error */
}
b = bzReadOpen ( &bzerror, f, 0, NULL, 0 );
if (bzerror != BZ_OK) {
   bzReadClose ( &bzerror, b );
   /* handle error */
}

bzerror = BZ_OK;
while (bzerror == BZ_OK && /* arbitrary other conditions */) {
   nBuf = bzRead ( &bzerror, b, buf, /* size of buf */ );
   if (bzerror == BZ_OK) {
      /* do something with buf[0 .. nBuf-1] */
   }
}
if (bzerror != BZ_STREAM_END) {
   bzReadClose ( &bzerror, b );
   /* handle error */
} else {
   bzReadClose ( &bzerror );
}

Utility functions

bzBuffToBuffCompress

   int bzBuffToBuffCompress( char*         dest,
                             unsigned int* destLen,
                             char*         source,
                             unsigned int  sourceLen,
                             int           blockSize100k,
                             int           verbosity,
                             int           workFactor );

Attempts to compress the data in source[0 .. sourceLen-1] into the destination buffer, dest[0 .. *destLen-1]. If the destination buffer is big enough, *destLen is set to the size of the compressed data, and BZ_OK is returned. If the compressed data won't fit, *destLen is unchanged, and BZ_OUTBUFF_FULL is returned.

Compression in this manner is a one-shot event, done with a single call to this function. The resulting compressed data is a complete bzip2 format data stream. There is no mechanism for making additional calls to provide extra input data. If you want that kind of mechanism, use the low-level interface.

For the meaning of parameters blockSize100k, verbosity and workFactor,
see bzCompressInit.

To guarantee that the compressed data will fit in its buffer, allocate an output buffer of size 1% larger than the uncompressed data, plus six hundred extra bytes.

bzBuffToBuffDecompress will not write data at or beyond dest[*destLen], even in case of buffer overflow.

Possible return values:

      BZ_PARAM_ERROR 
         if dest is NULL or destLen is NULL
         or blockSize100k < 1 or blockSize100k > 9
         or verbosity < 0 or verbosity > 4 
         or workFactor < 0 or workFactor > 250
      BZ_MEM_ERROR
         if insufficient memory is available 
      BZ_OUTBUFF_FULL
         if the size of the compressed data exceeds *destLen
      BZ_OK 
         otherwise

bzBuffToBuffDecompress

   int bzBuffToBuffDecompress ( char*         dest,
                                unsigned int* destLen,
                                char*         source,
                                unsigned int  sourceLen,
                                int           small,
                                int           verbosity );

Attempts to decompress the data in source[0 .. sourceLen-1] into the destination buffer, dest[0 .. *destLen-1]. If the destination buffer is big enough, *destLen is set to the size of the uncompressed data, and BZ_OK is returned. If the compressed data won't fit, *destLen is unchanged, and BZ_OUTBUFF_FULL is returned.

source is assumed to hold a complete bzip2 format data stream. bzBuffToBuffDecompress tries to decompress the entirety of the stream into the output buffer.

For the meaning of parameters small and verbosity, see bzDecompressInit.

Because the compression ratio of the compressed data cannot be known in advance, there is no easy way to guarantee that the output buffer will be big enough. You may of course make arrangements in your code to record the size of the uncompressed data, but such a mechanism is beyond the scope of this library.

bzBuffToBuffDecompress will not write data at or beyond dest[*destLen], even in case of buffer overflow.

Possible return values:

      BZ_PARAM_ERROR 
         if dest is NULL or destLen is NULL
         or small != 0 && small != 1
         or verbosity < 0 or verbosity > 4 
      BZ_MEM_ERROR
         if insufficient memory is available 
      BZ_OUTBUFF_FULL
         if the size of the compressed data exceeds *destLen
      BZ_DATA_ERROR
         if a data integrity error was detected in the compressed data
      BZ_DATA_ERROR_MAGIC
         if the compressed data doesn't begin with the right magic bytes
      BZ_UNEXPECTED_EOF
         if the compressed data ends unexpectedly
      BZ_OK 
         otherwise

Using the library in a stdio-free environment

Getting rid of stdio

In a deeply embedded application, you might want to use just the memory-to-memory functions. You can do this conveniently by compiling the library with preprocessor symbol BZ_NO_STDIO defined. Doing this gives you a library containing only the following eight functions:

bzCompressInit, bzCompress, bzCompressEnd
bzDecompressInit, bzDecompress, bzDecompressEnd
bzBuffToBuffCompress, bzBuffToBuffDecompress

When compiled like this, all functions will ignore verbosity settings.

Critical error handling

libbzip2 contains a number of internal assertion checks which should, needless to say, never be activated. Nevertheless, if an assertion should fail, behaviour depends on whether or not the library was compiled with BZ_NO_STDIO set.

For a normal compile, an assertion failure yields the message

   bzip2/libbzip2, v0.9.0: internal error number N.
   This is a bug in bzip2/libbzip2, v0.9.0.  Please report
   it to me at: jseward@acm.org.  If this happened when
   you were using some program which uses libbzip2 as a
   component, you should also report this bug to the author(s)
   of that program.  Please make an effort to report this bug;
   timely and accurate bug reports eventually lead to higher
   quality software.  Thx.  Julian Seward, 27 June 1998.

where N is some error code number. exit(3) is then called.

For a stdio-free library, assertion failures result in a call to a function declared as:

   extern void bz_internal_error ( int errcode );

The relevant code is passed as a parameter. You should supply such a function.

In either case, once an assertion failure has occurred, any bz_stream records involved can be regarded as invalid. You should not attempt to resume normal operation with them.

You may, of course, change critical error handling to suit your needs. As I said above, critical errors indicate bugs in the library and should not occur. All "normal" error situations are indicated via error return codes from functions, and can be recovered from.

Making a Windows DLL

Everything related to Windows has been contributed by Yoshioka Tsuneo
(QWF00133@niftyserve.or.jp / tsuneo-y@is.aist-nara.ac.jp), so you should send your queries to him (but perhaps Cc: me, jseward@acm.org).

My vague understanding of what to do is: using Visual C++ 5.0, open the project file libbz2.dsp, and build. That's all.

If you can't open the project file for some reason, make a new one, naming these files: blocksort.c, bzlib.c, compress.c, crctable.c, decompress.c, huffman.c,
randtable.c and libbz2.def. You might also need to name the header files bzlib.h and bzlib_private.h.

If you don't use VC++, you may need to define the proprocessor symbol _WIN32.

Finally, dlltest.c is a sample program using the DLL. It has a project file, dlltest.dsp.

I haven't tried any of this stuff myself, but it all looks plausible.

Go to the first, previous, next, last section, table of contents.